User interface for streaming spoken query

ABSTRACT

Methods and systems for information retrieval include analyzing audio data to produce word hypotheses. Displaying the word hypotheses in motion at different respective speeds at once across a graphical display. Information is retrieved in accordance with one or more selected terms from the displayed word hypotheses.

BACKGROUND Technical Field

The present invention relates to user interfaces for informationretrieval and, more particularly, to information retrieval based onautomated speech recognition and user guidance.

Description of the Related Art

As users conduct conversations, for example over the phone or with voiceor video chat software, the need will often arise to access a documentthat is pertinent to the discussion. Users may have difficulty findingsuch documents while simultaneously conducting their conversation,particularly if it necessitates switching to a different window orsystem and covering the video component of a video chat.

In addition, automating the retrieval of information based on aconversation is difficult. For example, automated speech recognition hasdifficulty determining correct search terms from human speech, which mayresult in the correct information being overlooked. As such, there areno adequate solutions for automated information retrieval responsive toa real-time conversation.

SUMMARY

A method for information retrieval includes analyzing audio data toproduce word hypotheses. Displaying the word hypotheses in motion atdifferent respective speeds at once across a graphical display.Information is retrieved in accordance with one or more selected termsfrom the displayed word hypotheses.

A system for information retrieval includes a speech recognition modulethat includes a processor configured to analyze audio data to produce aplurality of word hypotheses. A graphical user interface is configuredto display the plurality of word hypotheses in motion at differentrespective speeds at once across a graphical display. An informationretrieval module is configured to retrieve information in accordancewith one or more selected terms from the displayed word hypotheses.

These and other features and advantages will become apparent from thefollowing detailed description of illustrative embodiments thereof,which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The disclosure will provide details in the following description ofpreferred embodiments with reference to the following figures wherein:

FIG. 1 is a block/flow diagram of an information retrieval method inaccordance with the present principles;

FIG. 2 is a diagram of a graphical user interface in accordance with thepresent principles; and

FIG. 3 is a block diagram of a spoken query information retrieval systemin accordance with the present principles.

DETAILED DESCRIPTION

Embodiments of the present invention use automated speech recognition(ASR) to generate a list of the n best matches in a real-timeconversation to use as keywords. A user interface then provides a visualscrolling interface of all of the keywords, allowing a user to selectone or more of the keywords to use for information retrieval.

Using real-time ASR, the results of conversations can be used as inputfor information retrieval queries to retrieve documents relative to theconversation. For example, automatic retrieval of a particular manual orfrequently asked questions would be particularly useful for a techsupport call center agent, who would then not need to manually call upthe information.

ASR generally operates by producing a list of possible transcriptionsfor a given word and finding the most probable match. One challenge inusing the most likely match, however, is that recognition errorsnevertheless frequently occur. An information retrieval query that isbased solely on the most likely match is likely to have an unacceptablyhigh rate of failure, as the system would frequently fail to find amatch or would return erroneous results.

To address this problem, the present embodiments provide all of the topn results of ASR to the user. Using all of the results to performinformation retrieval increases the likelihood of finding the correctdocument(s), but does so at the expense of also producing a large numberof irrelevant documents due to the greater number of recognition errorspresent in these results.

One possible solution to this problem is to present all of therecognized keywords on the screen at once and allow the user to selectwhich options they may find most appropriate. However, showing all ofthe keywords in the n best results will quickly fill the screen withkeywords, making it difficult to find the correct keywords to select toreach a particular kind of data.

In one particular embodiment, these possible matches are provided as ascrolling interface, where keywords move across the screen as theconversation goes on, giving the user the opportunity to select one ormore of the keywords for information retrieval. The present embodimentsthereby combine automatic and manual elements, allowing the user toguide the automatic speech recognition and information lookup byselecting specific options out of the automatically generated set.

Referring now to the drawings in which like numerals represent the sameor similar elements and initially to FIG. 1, an information retrievalmethod is shown. Block 102 captures streaming speech data from one ormore sensors. It is specifically contemplated that the information maybe collected from a microphone at the user's position, from a microphoneat the position of the person the user is talking to, from informationtransmitted between the users, or some combination of the above. It isparticularly contemplated that speech data may be captured directly froma computer microphone and combined with speech data received from astreaming speech service or directly from another user.

Block 104 performs ASR to process the speech data and to produce, foreach spoken word, the best n ASR results, representing differentpossible interpretations of the word. Each result is assigned arespective confidence score that reflects the ASR process's confidencethat the result in question is the correct interpretation of the speechdata. In one exemplary embodiment, each result may be assigned apercentage score that represents the confidence in the result relativeto the other results.

Block 106 assigns an importance score to each of the results. Thisimportance score is distinct from the confidence score and represents alikelihood that the result in question is relevant to information thatthe user may want to retrieve. To use a trivial example, if the userspeaks the word, “the,” this word is unlikely to produce usefulinformation retrieval results and would thus have a low importancescore. On the other hand, if the user speaks a specific product name,brand name, or other identifying information, this is quite likely to berelevant to the discussion and will have a high importance score. Block108 then removes any results having an importance score that is lowerthan an importance threshold. This reduces the number of results to keepthe user's interface from becoming too cluttered.

Block 110 displays the remaining results on a graphical user interface.The size of the displayed results corresponds to the importance score ofeach respective result. In this way, terms that are more likely to berelevant are emphasized, both in the user's attention and in the user'sability to easily interact with them. Block 112 moves the displayedresults across a screen area with a speed that has an inverserelationship to each result's confidence score. In other words, as theconfidence score increases, results are scrolled across the screen at aslower speed, giving the user more time to interact with them beforethey disappear. Block 112 also handles the removal of results from thescreen when they reach a far edge of the screen area.

After a user selects one or more of the results from the screen, whetherby clicking using a pointer, through a key combination, or some otherform of human-computer interaction, block 114 performs a search forinformation that relates to the user's selection. This search may be,for example, a search on a search engine, a search through a private orpublic archive, a database query, or some other structured orunstructured information retrieval operation. The information retrievedby block 114 may be presented to the user via, e.g., a graphical userinterface, by saving the information locally, by printing a document tohardcopy form, etc.

The present embodiments solve the problem of tokenization boundarymismatch between ASR and information retrieval systems. Tokenizationboundary mismatch arises when the ASR process divides terms into chunksin a way that does correspond well to the way those terms are written,such that an information retrieval search based solely on the ASRresults will not find the correct information. In one example, if a userspeaks the term, “HTML5,” the ASR process may mistakenly transcribe thisas two separate terms, “HTML,” and the word, “five.” Performing a searchbased on these terms separately, or even using both terms but with thenumeral spelled out, will produce inaccurate results. For results whichhave multiple different tokenizations, those different options should bedisplayed on the user interface to allow the user to select the mostappropriate term.

Referring now to FIG. 2, a diagram of an exemplary user interface isshown. The user interface includes three main regions: a scroll area210, a selection stack 212, and a search button 214. The scroll area 210displays a set of results, each of which has a font size and a scrollspeed set in accordance with its importance and confidence scores. Inthis diagram, the lengths of the arrows indicate a speed and are notintended to represent visual design elements. A first result 202 shows aresult having an average importance and confidence. A second result 204has a lower importance and confidence—thus its font size is smaller andthe arrow representing its speed is longer. A third result 206 has veryhigh importance and confidence—thus its font size is large and its speedis lower. Although the embodiment shown has only horizontal motion fromright to left, it is contemplated that any combination of orientationand direction may be employed. It is also contemplated that non-linearpaths may be used if appropriate. In addition, other visual identifiers,such as color, may be used to group and emphasize different results,with high-contrast colors being used to emphasize particularly usefulresults.

Generally the interface will attempt to put results onto differenthorizontal paths, to minimize interference between different results.However, in a situation where many results are available at once (forexample, in a rapid conversation that has a high number of keywords),the scroll area 210 may include multiple results on a same horizontalpath. In one example, the second result 204 and a fourth result 208 areon overlapping horizontal paths. In this case, the interface determinesthat the fourth result 208 will finish its trip across the scroll area210 before the second result 204 reaches its position, thereby ensuringthat there will be no collision. In addition, new, slow-moving resultcan always be placed on a same horizontal path as an existing,faster-moving result without a risk of collision. By increasing the sizeand decreasing the speed of keywords with high importance and highconfidence, the interface makes it easier for the user to click thosekeywords. Meanwhile, the other keywords are still shown, allowing theuser to still select them if the ASR process was incorrect in itsestimates.

In one particular embodiment, results that come from the same portion ofa conversation are associated with one another and may be spatiallygrouped together. In this embodiment, when the user selects one resultfrom a set of mutually associated results, the speed and size of theother associated results may also be adjusted, making it easier toselect those associated words. Words may be associated with one anotherif they are spoken closely together in time or if there is some semanticor ontological relationship between them. For example, if a userdescribes a product, the product name, the brand name, and anydescriptive identifiers may be taken together as associated results,allowing the user to easily build a query based on the full phrase.Color may also be used to help emphasize this grouping, with multipleresults that are within the same associated set sharing a color.

In contrast, a result may be a member of a set of competing hypotheses.If, for example, multiple different tokenizations are possible for agiven result and the user selects one of them, the other hypotheses maybe de-emphasized and made to disappear faster. In one case, thecompeting hypotheses may be removed immediately. In another case, thecompeting hypotheses may be graphically de-emphasized by reducing theirsize or changing their color to be lower contrast or grayed-out.

The selection stack 212 includes a list of results that have beenselected by the user. As the results move by in the scroll area 210, theuser can interact with one or more of them by, e.g., clicking on themwith an interface device or by using a keyboard shortcut. In the case ofa keyboard shortcut, the results may be numbered, allowing the user toselect results by pressing a corresponding number key. After a resulthas been selected, it is listed in the selection stack 212. Any numberof results may be so selected to be combined into a single informationretrieval query. By selecting the search button 214 the user triggersthe query based on the results in the selection stack 212.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Reference in the specification to “one embodiment” or “an embodiment” ofthe present principles, as well as other variations thereof, means thata particular feature, structure, characteristic, and so forth describedin connection with the embodiment is included in at least one embodimentof the present principles. Thus, the appearances of the phrase “in oneembodiment” or “in an embodiment”, as well any other variations,appearing in various places throughout the specification are notnecessarily all referring to the same embodiment.

It is to be appreciated that the use of any of the following “/”,“and/or”, and “at least one of”, for example, in the cases of “A/B”, “Aand/or B” and “at least one of A and B”, is intended to encompass theselection of the first listed option (A) only, or the selection of thesecond listed option (B) only, or the selection of both options (A andB). As a further example, in the cases of “A, B, and/or C” and “at leastone of A, B, and C”, such phrasing is intended to encompass theselection of the first listed option (A) only, or the selection of thesecond listed option (B) only, or the selection of the third listedoption (C) only, or the selection of the first and the second listedoptions (A and B) only, or the selection of the first and third listedoptions (A and C) only, or the selection of the second and third listedoptions (B and C) only, or the selection of all three options (A and Band C). This may be extended, as readily apparent by one of ordinaryskill in this and related arts, for as many items listed.

Referring now to FIG. 3, a spoken query information retrieval system 300is shown. The system 300 includes a hardware processor 302 and a memory304. In addition, the system 300 includes a graphical user interface 306and an input device 308. The graphical user interface 306 may includeone or more displays to present visual information to the user. Theinput device may include one or more of a pointer device, a keyboard, amicrophone, and any other type of input mechanism.

The system 300 includes a set of functional modules. These modules maybe implemented as software that is executed on the hardware processor302 or may, alternatively, be implemented individually or together asdiscrete hardware components in the form of, e.g., application specificintegrated chips or field programmable gate arrays.

In particular, the system includes a speech recognition module 310 thatprocesses audio speech data and creates a set of hypotheses as to whatwords the speech data represents. The audio speech data may be accessedfrom memory 304 or may, alternatively, by streamed directly to thespeech recognition module 310 from the input device 308. A controlmodule 314 selects a set of the n best hypotheses, judged by anassociated confidence score, and displays them on the graphical userinterface 306 as described above.

The user selects one or more of the results using the input device 308.The control module 314 may trigger further refinements of the displayedresults in the graphical user interface 306 based on user selections tomake it easier for the user to select the desired results. After havingselected all of the results the user desires, the user initiates aninformation retrieval query using the input device 308. The controlmodule 314 passes this query to information retrieval module 312. Theinformation retrieval module 312 in turn executes the query, either bysearching for matching documents stored in local memory 304 or byaccessing one or more external data stores, such as search engines, tofind pertinent information. The control module 314 then provides theretrieved information to the graphical user interface 306 for display tothe user.

It should be recognized that, by providing information by theinformation retrieval query, the present embodiments materially affectthe lives of users by speeding the process of accessing thatinformation. For example, locating such information manually may be verytime consuming and distracting, detracting from the user's conversation.The above embodiments provide distinct advances in data management,query handling, and searching. In addition, data retrieval is a basicfunction of a computer, and so an improvement in query handling for dataretrieval represents an improvement in the functioning of the computeritself.

Having described preferred embodiments of a user interface for streamingspoken query (which are intended to be illustrative and not limiting),it is noted that modifications and variations can be made by personsskilled in the art in light of the above teachings. It is therefore tobe understood that changes may be made in the particular embodimentsdisclosed which are within the scope of the invention as outlined by theappended claims. Having thus described aspects of the invention, withthe details and particularity required by the patent laws, what isclaimed and desired protected by Letters Patent is set forth in theappended claims.

What is claimed is:
 1. A method for information retrieval, comprising:analyzing audio data using a processor to produce a plurality of wordhypotheses; displaying the plurality of word hypotheses in motion atdifferent respective speeds at once across a graphical display, thespeeds being based on a confidence value associated with each respectiveword hypothesis; and retrieving in accordance with one or more selectedterms from the displayed word hypotheses.
 2. The method of claim 1,wherein each of the plurality of word hypotheses has an associatedimportance value.
 3. The method of claim 2, wherein only word hypotheseshaving an importance above an importance threshold are displayed.
 4. Themethod of claim 2, wherein displaying the plurality of word hypothesescomprises displaying each word hypothesis at a larger size for largerimportance values.
 5. The method of claim 1, wherein analyzing audiodata comprises associating sets of related and competing hypotheses. 6.The method of claim 5, wherein displaying the plurality of wordhypotheses comprises visually emphasizing other hypotheses in a set ofrelated hypotheses after a user selects a first hypothesis from the setof related hypotheses.
 7. The method of claim 5, wherein displaying theplurality of word hypotheses comprises visually deemphasizing otherhypotheses in a set of competing hypotheses after a user selects a firsthypothesis from the set of competing hypotheses.
 8. A non-transitorycomputer readable storage medium comprising a computer readable programfor information retrieval, wherein the computer readable program whenexecuted on a computer causes the computer to perform the steps of:analyzing audio data using a processor to produce a plurality of wordhypotheses; displaying the plurality of word hypotheses in motion atdifferent respective speeds at once across a graphical display, thespeeds being based on a confidence value associated with each respectiveword hypothesis; and retrieve in accordance with one or more selectedterms from the displayed word hypotheses.
 9. A system for informationretrieval, comprising: a speech recognition module comprising aprocessor configured to analyze audio data to produce a plurality ofword hypotheses; a graphical user interface configured to display theplurality of word hypotheses in motion at different respective speeds atonce across a graphical display, the speeds being based on a confidencevalue associated with each respective word hypothesis; and aninformation retrieval module configured to retrieve information inaccordance with one or more selected terms from the displayed wordhypotheses.
 10. The system of claim 9, wherein each of the plurality ofword hypotheses has an associated importance value.
 11. The system ofclaim 10, wherein only word hypotheses having an importance above animportance threshold are displayed.
 12. The system of claim 10, whereinthe graphical user interface is further configured to display each wordhypothesis at a larger size for larger importance values.
 13. The systemof claim 9, wherein the speech recognition module is further configuredto associate sets of related and competing hypotheses.
 14. The system ofclaim 13, wherein the graphical user interface is further configured tovisually emphasize other hypotheses in a set of related hypotheses aftera user selects a first hypothesis from the set of related hypotheses.15. The system of claim 13, wherein the graphical user interface isfurther configured to visually deemphasize other hypotheses in a set ofcompeting hypotheses after a user selects a first hypothesis from theset of competing hypotheses.